Lost Purposes

My head kinda aches. I think maybe I'll go watch some TV.

We agree that an individual can manage their goal better - keeping an eye on the big picture, not ending up with lost purposes - than an organization. But what are you proposing (are you proposing anything?) Should we simplify, get rid of large and complex organizations? Seems that even when you count all the waste of lost purposes, we're more productive with them, at least in many cases ("better off" is another question, but let's leave that aside since it doesn't really bear directly on this issue). Complexity and scale have advantages, as well as this disadvantage; the mere existence or even extent of "lost purposes" doesn't imply lower overall efficiency.

When I try to explain to a computer how to do what I want (programming), my first explanation always leads to a million lost purposes (bugs). I test it, find the bugs, and clarify my instructions, trying to make them less ambiguous - less possible for the computer to fulfil the letter of my law without the spirit. Eventually I reach a point where testing doesn't turn up many more problems, but of course I'll never eliminate all lost purposes - software of sufficient complexity, even the most vetted commercial software, still has loads of bugs (and even aside from bugs, times where it follows the letter rather than the spirit of what the programmer intended). So why do we bother? Because the formalized instructions are more leverageable, and the computer is faster, than doing the work myself. Faster by enough orders of magnitude to make up for all the lost purposes and then some.

Ultimately my takeaways are:

-We should create only the laws/organizations/programs that make us much better off, even taking into account all the pitfalls of lost purposes.

-We should design these laws/organizations/programs as carefully as possible, with a constant eye to how every intention we express can lead to a frustrated goal.

-We should test and reassess these laws/organizations/programs as often as we possibly can, in every way that we possibly can, to catch as many instances as possible of their failing to accomplish what we wanted, and thus to continually refine them. And we should take the testability and refinability of a mechanism into account, when deciding whether it's worthwhile. (Computer programs are particularly amenable to this kind of testing, which is why they generally end up doing about 80% of what they're supposed to; an education system is much harder to treat this way.)

I think the concern of lost purposes is often underweighed by actual policymakers, but I also think it's unproductive to complain about lost purposes in the abstract, and misguided to design our lives around avoiding them (rather than avoiding the cases where they outweigh their benefits). Incorporating concerns about them (in the ways listed above) seems useful.

The general point is valid, but in particular cases one should be cautious in concluding than an org has lost it purpose - it might have a different purpose than you realized. School in particular my not have the purpose to teach, but to sort students by abilities.

In cases of delayed effects, linking pensions to outcomes years later might provide incentives.

For instance, teachers pensions could depend on the eventual earnings of their pupils, or elections could have tickboxes for evaluating the last politician.

(which doesn't help the problem of credit being due to many people)

Great post again!! This seems to be a good portal to the series as well! But my gosh I feel infoleptic!! I feel as though I would need 100 more brains to be able to do and read and learn everything I would like to with 1 brain! And if I would have a 100 brains I would probably feel I would need a 1000! etc. I think it is really sad this blog didn´t have some kind of rating system! There seems to be so many good posts. And if someone totally new to this would come here it would be hard to know where to start. So a list of the most important posts would be really great! Although 80% of your stuff seems to always hold the your standards! It would also be good if there would be a way to browse only your posts. Not that the other posts aren´t good, it is just that your posts have certain themes which one would like to hold on to and go deep with!

I am a great believer in utilizing/abusing our social brain for pursuing our higher values! Have you ever thought about having some kind of minimal microblog / status indicator of your real work on AI. Of course this would take some minutes of your precious time. This could make your work feel more connected to the world of other beings, which this is ultimately aimed for. If you would not have done any progress in a week you would directly feel embarrassed in front of humanity(which I am sure you already do, and I do too). If you would make great progress you would feel proud, and people could see it and give you emotional credit! I know you may not need this kind of system, you seem to have a very robust and disciplined mental operating system. It may also be that it would be only distracting to your real intrinsic altruism, or even distort your work. But it may also be that it would act as a healthy pusher, and make it more explicit that this is not your private project but the greatest project of humanity, which you happen to be one of the few that have competence to actually work on. It could also act as a please for people to come up with stimulating ideas(i.e. bad ideas that stimulate better ideas). But you already have a lot of distraction, so the balance have to be hard to keep. Good luck! Keep on pushing!

As always, well put. However, "Maybe you realized it was all madness, but I bet you did it anyway. You didn't have a choice, right?" Many students do have a terribly mercenary attitude. But a good number do voluntary study topics with no evident financial incentive.

"...I didn't need to know what the words meant, it would work anyway." That was a stupid thing for them to say. However, a common approach in these circumstances, similar but crucially different is "you don't need to know what the words mean now, just read them over and over as you gradually develop your understanding -- it's a good way to learn."

"..classroom time is now spent on preparing for tests..." Is there a better way of teaching, as proven by high-quality studies? (There may well be, but the question must be asked before dismissing the test-based approach.)

Not to get too sidetracked, because your overall point about education is well-taken, but this:

A recent study here in the Bay Area showed that 80% of teachers in K-5 reported spending less than one hour per week on science, and 16% said they spend no time on science. Why? I'm given to understand the proximate cause is the No Child Left Behind Act and similar legislation. Virtually all classroom time is now spent on preparing for tests mandated at the state or federal level. I seem to recall (though I can't find the source) that just taking mandatory tests was 40% of classroom time in one school.

is implausible on its face (particularly the assertion about 40% of classroom time -- that would mean kids spent 2+ hours every day taking tests). And if schools are ditching science (really? they're really spending 3 hours a day just on reading skills and 3 hours a day just on math?), that would be a very foolish decision on their part, even if their goal is to show high reading scores. Once kids know how to decode words, the best path to reading comprehension is lots and lots of substantive background knowledge -- such as science, history, etc.

By the way, the paragraphs about opening and shutting the car door are one of the best things I've ever read about the distortion of incentives in organizations.

If they have no interest in knowing the information for its own sake, that sounds like a problem with them, not with the university.

Not everyone might be familiar with the expression "K-5". I had to google for it:

K-5 (pronounced "kay through five") is an American term for the education period from kindergarten to fifth grade.

If you think about this for a while, you'll begin to realize how much of our civilization is based on lost purposes.

Why do men wear ties? Why do we build houses out of the one substance that rots and burns? Why do we type with QWERTY keyboards? Why isn't the department of defense also responsible for homeland security (what does defense mean, if not homeland security)?

There is an interesting bias in the way that we approach these issues. We see ourselves as the prisoners of the system not the jailors. Or maybe I should say that we don't like to admit that we are lords as well as serfs.

One political response to the problem of lost purposes is to focus on government coercion which leads to libertarianism. I don't think that we should rush so quickly to a conclusion. First we should examine the origins of the problem more closely.

The key concept is contestability: is there an alternative BA course you can take that does not require mastery of 12 century knitting patterns. This not lead automatically to free-marketry. One can image contestible socialism.

Think about driving licences. Why does there have to be a single agency that issues them. The government could fund two, splitting the money in proportion to the number of licences issued. If queues are too long at one office, drivers will go to the over. It gets rather interesting if the competing offices get to set different driving tests. Do you go for the easy one or the hard one. Perhaps drivers who have passed the hard one get cheaper insurance rates. Perhaps that also tells you something important about the extra preparation being personally worth while, not just a money saving measure.

My favourite example is prison regimes. What do we expect from prison? We could set up a Prison Audit Agency that follows up prisoners 5 years after release to see how they are earning their money. There are many possibilities: professional criminal, dead, junkie, wino, deadbeat, honest employee, honest sole-trader, honest employer.

Imagine that the rehabilitation figures got the same kind of political attention as the crime figures. It would open the door to randomised assignment to different regimes. The hangers and floggers could run one prison, the hugs and therapy crowd another.

My guess is that neither really knows much about rehabilitation, a point that would become very clear a decade on as the figures are disappointing for both regimes. Both groups would be humbled and forced to think harder about what they were doing least their rivals responded more wisely and eventually built up a commanding lead in the results.

School vouchers are another example. Most state supported school systems are designed as pay-twice systems. If you pull your child out, you continue to pay taxes as before, but do not get the money back. You must find a second tranche of funds to pay your child's school fees. So schooling is only weakly constestible.

The curious point is that state power is weakly contestible, leading to poor delivery of services to the voter, who wearies of his political leaders and votes them out of office. Why do politicians favour a system that costs them office?

As we follow politics we see that reforms that make public services contestible, even when they remain in the state sector, are bitterly contested by civil servants. And not just public services. Trade barriers are about stopping the consumer from buying Toyota instead of General Motors, because the auto workers don't want their claim to be building wonderful cars to be contestible.

The 90% of us who are above average will do anything to keep the sunshine of contestibility from illuminating our particular patch line of work and causing our illusion to crumble to dust.

I don't see any hope for a direct attack on our tendency to lose our purposes. We like it that way and will fight to keep it like that. More insidiously we will always see ourselves as serfs not lords. Is there a name for this bias, the one that stops us admitting to ourselves that we are doing it to ourselves?

As someone who actually practices medieval fighting arts I have to take objection to this:

Miyamoto Musashi said:

"The primary thing when you take a sword in your hands is your intention to cut the enemy, whatever the means."

This is of course a generalization. The reality is more complex. Never forget that the real primary goal of yours is to stay alive and unharmed(at least that's my goal), so sometimes a parry makes sense, even if you can't cut the enemy in the same movement. You could cut or kill the enemy a lot of times if you simply ignore his blade and go straight for the attack/counter-attack, but that would also mean that you would be killed or injured.

Btw you don't want to always cut the enemy. Sometimes your objective will be to disarm him and take him captive without seriously injuring him.

But in GENERAL his advice is correct: defend and attack(cut) in the same movement, that's a good guideline although it's not always possible to do that depending on the actual situation.

Btw generating the entire art from withhin yourself would be kinda hard like trying to generate physics from yourself, you better have a good teacher. Of course you probably mean that you should be able to understand the idea behind each move so if the teacher doesn't explain it you better think about it and try to figure it out for yourself.

I don't understand your use of the word "versus" in the following sentence:

"The corresponding essential of pragmatic rationality - decision theory, versus probability theory - is to always see..."

versus means: As the alternative to or in contrast with.

I thought that probability theory is an integral part of decision theory, so the "versus" doesn't make sense.

A brilliant post with many links to the Yudkowsky Canon. It has just become a bookmark.

One quip: the study which revealed that a majority of research findings were false seemed to rely on a simulation, and on one meta-study performed earlier by the group. Have I understood this correctly?

Perhaps the p for biological experiments should be lower, but my first inclination is to defend the field I work in and its custom of p<0.05 .

Every time I open up an animal for surgery, the animals nerves are lying in slightly different places. There is a different amount of muscle-tissue obscuring the nerve I have to record from. I have to re-manufacture my electrode after every few surgeries. The hook on my electrode is differently shaped each time. When I put the nerve on the electrode, I pull it up with a different amount of force, in a different position, and do a different degree of damage to the nerve's processes.

Every animal is slightly different, and every surgery goes slightly differently.

Therefore, I think it might be understandable, why I need a p that is higher than a physicists, because if the physicist uses the same stock of materials, he has much less variability in his setup to begin with.

I find it ironic that a man who wants to abolish death is lecturing others about thorough research and being careful what goals you set.

Geremiah: it's worth understanding a problem before proposing a solution.

It seems to me there are plenty of 'sane' corporations out there, even very large ones. Eliezer-

Corporations have the singular purpose of making money, and its generally very simply to determine if and how well a given company is doing this. The problems you describe with our education system or Soviet shoe factories apply to a certain extent within a corporation itself, but if a company strays too much from its purpose of making money (starts going 'insane'), this becomes quickly apparent in the form of declining profits. And of course if it can't find a way to remedy this, the corporation goes bust. The end.

I guess I was just sort of surprised you mentioned corporations here, because the pressures in a free market tend to mitigate the sorts of problems you identify in public education and communist economies. Perhaps you had something else in mind when asking if corporations can be sane; If so, I'd like to hear it.

Stuart Buck, Why is the Barone quote the only vestige of the 93 stuff? What was the jist of that lunch?

As a high school dropout and aspiring philosopher of self-education, I salute you.

My son is homeschooled, and the homeschooling consists of... nothing. He studies spontaneously to solve problems that are authentic to him, just as I do. Mostly these involve video games and online fantasy role-playing games. It's like A.S. Neill's Summerhill school except he's all alone. There are disadvantages to this sort of education, but inauthenticity is not among them.

I am writing a book along these lines. It's about how we can creates ourselves as individual thinkers. It's called How I Learn Stuff: Secrets of a Buccaneer Scholar (old draft is temporarily online at http://www.satisfice.com/hils.pdf. The book itself should be finished in a few months).

Video -- I have no idea what you're talking about??

Stuart Buck, Apologies for the trollish behavior. I read your blog for a short while about 6 years back when I found that stuff intriguing, and seemed to recall it being the reason Barone contacted you. It makes perfect sense to erase that stuff (considering who was waving it around), but I found it interesting to find the Barone quote up there as a vestige. I'd love to know what he said about it all.

James Bach,

I am enjoying the online draft of your book very much, thank you for posting this!

I'm still not sure what you're getting at -- want to email me at stuartbuck@msn.com? Thanks.

Wow, Caledonian's comments are absolutely the worst...

mtc, you could make a few volumes of Dilbert your required reading, to inform your faith in the 'sane corporation'. In fact, in such high powered company, I'm surprised to see non-qualified discussion of the intentionality of any organisation. Organisations don't have intentionality, individuals do (perhaps... if we don't let the Evolutionary Psychology crowd take it away from us..). As all Eliezer's anecdotes illustrate, these get lost in the wash of large numbers and multiple levels, networks, and gridlocks, so we are left with the dross of the latest organisational mantra which everyone gives lip service to and no-one believes. It is trivial to observe that the dominant intentionalities will be those of the most powerful individuals, but these also are destined to be diluted and lost. However, my personal believe is that inadequate individuation (accepting cultural norms of need and intentionality) is a far bigger problem than such intentionality as we have getting lost. Let's first work out who we are and what we have to say before worrying about our voice getting lost.

Robin Hanson: School in particular my not have the purpose to teach, but to sort students by abilities.

Excusez moi, but what exactly is that supposed to mean? As far as I know, most principals want ALL of their students to go to Ivy League and make millions. I can't make any sense of your comment.

It means that contrary to his protestations, Robin really is a libertarian, and he is trying to push the meme that libertarian positions are really vanilla mainstream positions. Someone who is doing the exact opposite of overcoming bias, iow.

Tibba, when you are in a bar, do you see an attractive person and say to yourself, "I think I'll initiate sexualized body language, so that I can mate with that person, thereby increasing the frequency of my genes in future generations"?

There is another post addressing your incorrect objection, but I can't remember what it's called, maybe someone else can dig it up.

At Mt. Obaku temple in the Ko district, Yudkowksy-Sensei was approached by an Individualist during morning meditation.

"All of us are ultimately selfish; we care only about our own states of mind. The mother who claims to care about her son's welfare, really wants to believe that her son is doing well - this belief is what makes the mother happy. She helps him for the sake of her own happiness, not his." said the Individualist.

Yudkowsky-Sensei remained sitting on his zafu, and said nothing.

The Individualist continued: "She did it because she valued that choice above others - because of the feeling of importance she attached to that decision."

Yudkowksy-Sensei stood up and walked out of the temple to his car, where he proceeded to open and close the driver-side door several times before saying:

"There is no chocolate at the supermarket."

Caledonian: That didn't make any sense at all. But while we're finding things ironic, how about the fact that a man who loves death so much hasn't killed himself yet?

"Tibba, when you are in a bar, do you see an attractive person and say to yourself, "I think I'll initiate sexualized body language, so that I can mate with that person, thereby increasing the frequency of my genes in future generations"?"

Of course not. I want to get laid. That's my explicit desire. Promoting my genes is just a side effect of this. And what is the explicit desire of a principal? Perhaps to teach children. Perhaps to go to work, get paid and ensure that nobody sues him. (He'd still like it if they all go to Harvard.) But "sorting students by abilities" is so off the wall, I don't know how to iterpret it. Colleges and employers might see it that way, but is their opinion THAT important?

You don't have to have complex human organizations to get lost purposes. Runaway sexual selection is basically a subgoal of evolution (finding a good mate) taking over far too many resources from the primary goal (survival and reproduction). Of course, since this is probably the process that generated the human mind in the first place, we shouldn't judge the phenomenon too harshly. Our minds and culture are founded on distraction, which explains a lot, like why I am commenting here instead of working.

As for human organizations, I've always felt that, given that they are composed of people who all have their own goals, the real mystery or at least thing to be explained is how they sometimes actually fulfill their stated purpose rather than being constantly subverted or misdirected by the divergent goals of their component individuals. I suppose that the history of industry and management is the history of figuring out ways to stop this from happening.

It's not okay for organizations, corporations, or governments to be misdirected and focused on surface matters; but it's not uncommon, either. Musashi was unique for his insight -- with little training, he defeated masters from an early age, all because of clear perception. (He was later defeated by a man who, in a dream, saw that the jo staff (arm pit height staff) would allow him to defeat any man in Japan.) Perception like that is exceptional. A clear sense of purpose is a gift.

How does one cultivate this? The constantly feel the sharp distinction between success and failure -- as Musashi must have -- is one way. Are lives are gentler than that; I'm not sure what we moderns can do. Education does not seem to help.

mtc, you could make a few volumes of Dilbert your required reading The Logical Fallacy of Generalization from Fictional Evidence Better evidence for dysfunctional corporate governance is here and here.

It means that contrary to his protestations, Robin really is a libertarian, and he is trying to push the meme that libertarian positions are really vanilla mainstream positions. Someone who is doing the exact opposite of overcoming bias, iow. Robin Hanson has contrarian rather than vanilla mainstream positions, and is quite explicit about that. When he mentioned schools he didn't single out public schools (Eliezer, for example attended a Hebrew school).

Jason: AFAIK it's only Shindo Muso Ryu practitioners who claim that Muso Gonnosuke defeated Miyamoto Musashi. Is there any other documentation supporting this?

Excusez moi, but what exactly is that supposed to mean? As far as I know, most principals want ALL of their students to go to Ivy League and make millions. I can't make any sense of your comment.

http://en.wikipedia.org/w/index.php?title=Signalling_%28economics%29&oldid=165536682#Assumptions_and_groundwork

Dilbert isn't quite an example of "The Logical Fallacy of Generalization from Fictional Evidence": Scott Adams, in his non-fiction books, repeatedly mentions that no matter how crazy he makes his Dilbert strips, people keep telling him "That's just like my job." In other words, Dilbert is a documentary. ;)

Also, there's this book. If you want to know how corporations are actually run, you need to read it.

people keep telling him "That's just like my job." In other words, Dilbert is a documentary. ;)

Or a horoscope...?

Back on topic (sort of)... Some of the discussion here seems to highlight to problem of resolving dual and conflicting purposes. One key purpose of schools is to educate people; in a very real sense, though, they also necessarily serve to sort those of differing abilities (whether this is an overt purpose, pace Tiiba, is irrelevant: it is a function which schools are clearly set up to facilitate, and so is a de facto purpose). These two purposes conflict, at least to a degree, in that the process of streaming leads to those at the extreme ends of the bell curve being neglected.

At the top end - high achievers - the market can provide help: it makes sense to invest in talent. At the bottom end, some kind of intervention is required (at least is schools are to continue serving their first purpose of providing education). Hence interventions like NCLB - which seems to be failing as an intervention because it is itself a casualty of legislators' own dual and conflicting purposes... And so on it goes.

Tiiba,

I think Robin may have meant by "purpose" role in society rather than explicit intention of the principal. (In other words, the signaling is why employers continue to value education, and why people keep going to school.)

Similarly, even if you don't, for instance, go to a bar with the explicit purpose of propagating your genes, you are only naturally inclined to get laid because it's an evolutionarily fit behavior. This doesn't mean you can't explicitly hold another purpose in your mind.

I have seen one or two occasions when he has denied it. But I give more weight to actions, and I have not seen anything he has posted which runs contrary to that assumption. If he truly is a contrarian, you'd expect to see more posts arguing for points that are more commonly associated with socialism or communism. I haven't seen them.

To clarify, yes, by "purpose" I meant social function, not conscious intention, and yes I meant private schools as well as public.

A computer company won't hire you, no matter how much you know about 12th century knitting patterns, though. So even employers, AFAIK, value actual knowledge. And I don't think the purpose of a school should be determined from the point of view of companies. It should be seen according to those who actually determined whether it will be there - the senator who wrote it into the water bill, exc.

some young man or woman is sitting at a desk in a university, earnestly studying material they have no intention of ever using, and no interest in knowing for its own sake. They want a high-paying job, and the high-paying job requires a piece of paper, and the piece of paper requires a previous master's degree, and the master's degree requires a bachelor's degree, and the university that grants the bachelor's degree requires you to take a class in 12th-century knitting patterns to graduate. So they diligently study, intending to forget it all the moment the final exam is administered, but still seriously working away, because they want that piece of paper. Some people will tell me that Cuba has better schools than the USA, so I ask do they live better. What is the purpose of schooling? Do schooling and homework squeeze out learning, in net producing a less educated population? Alfie Kohn claims that homework does not help learning. Science and history are evidently fun to many people, as evidence I point out the existence of the science, discovery and history channels. So does homework sqeeze out learnig useful science?

Is school just a long test? If so are home schoolers and those who hire tutors for their children cheating the rest of society?

@Tiiba:

The choices that create a school are not the only choices responsible for the schooling that goes on. Students choose to attend, or others choose for them (parents, gov't, etc.). They do so in response to incentives created by others (businesses, higher educational institutions, etc.).

And of course employers value some kinds of knowledge. But knowledge and schooling are not the same thing. At least, people behave w/ regard to school in ways I can't imagine their behaving if it were just about knowledge (e.g. caring about grades, fulfilling major and core requirements that they don't want to but need in order to graduate).

This doesn't mean that school never accomplishes anything beyond signaling, and I suspect Robin Hanson doesn't believe that either. But often, its major role is as a signaling mechanism.

Part of the thinking behind NCLB and similar policies is the focus on measurability/transactionalization. One of underlying ideas is: if something can't be easily modeled/measured, it's not to be considered. The corollary to that is that things that are easily measured (test scores, in this case) are piled on, sometimes thoughtlessly.

http://www.huffingtonpost.com/charles-h-green/software-programming-and-_b_69287.html

is a brief, but enlightening article on the subject.

mtc has it right in that the difference between private organisations and state (and state-funded) ones is that private organisations have an in-built feedback mechanism, they exist to make a profit, the profit they make is calculable by the owners, who will either keep their organisation profitable ie true to its purpose, or sell out or go bust. State organisations by contrast may not have a clear purpose, often find that purpose diverted or confused by politicians, have no feedback mechanism, and almost never go out of existence for any reason including gross failure.

or sell out or go bust

...or, if their function is sufficiently essential, get bailed out by the state.

"..classroom time is now spent on preparing for tests..."

Sounds a bit like learning.

NCLB is just a way for the Feds to see if they are getting their moneys worth. They spend a lot per student and should know if the student is learning anything. It costs the school one day a year for the test. Everything on the test is what the student is supposed to be learning anyway.

Believe me, preparing for a standardized test is VERY DIFFERENT from actually learning and understanding material.

Depends who you are, doesn't it? I think a lot of us privileged folk shudder at the idea of "teaching to the test," because we went to schools were everyone was so far above the standard that this would have been dull and stunting. But NCLB doesn't exist to help high-proficiency students, and I imagine it has next to no effect on them. The classrooms that need to "teach to the test" are the ones that have trouble passing it, and they generally consist of students whose main problem is that they can't read or can't do basic arithmetic.

I haven't seen the tests myself, but it actually seems like you'd have to be quite incompetent to design a test on reading that a child can pass without being able to read. Bottom line, test scores don't seem like much of a "lost purpose" when we're talking about underperforming schools (and giving up some teacher autonomy/creativity is, frankly, very likely not a cost in these cases).

I think a lot of us privileged folk shudder at the idea of "teaching to the test," because we went to schools were everyone was so far above the standard that this would have been dull and stunting.

Um... that seems rather implausible. I went to perfectly normal and average schools, and they were already dull and stunting without "teaching to the test".

You just written a lot of words, it could be better to write a substract or bolden your main sentences

Caledonian, if I understand Holden correctly, he's just arguing that many people here may have had an untypical educational experience.

Both my parents are veteran educators (teachers, then administrators) in a tough urban school district in the Midwest, and they both laud No Child Left Behind not because it helps children directly, but because it's the only "objective" information available that can be politically wielded to attack known underperforming schools and employees. Administrators are often too well aware how terrible their schools, teachers, and students are, and NCLB testing doesn't tell them anything that they didn't already know. Instead, the tests are acceptable "objective measurements" to blame to make decisions that the administration knew had to be made anyway. That way, it's "the test's fault" instead of the fault of the administrator. Before NCLB, there was nothing to blame, and so hard decisions couldn't be made because it was politically impossible to do so.

As some guy once said, small is beautiful.

http://en.wikipedia.org/wiki/Small_is_Beautiful

Why would any individual lose track of their purposes in a swordfight? If someone else had taught them to fight, if they had not generated the entire art from within themselves, they might not understand the reason for parrying at one moment, or springing at another moment; they might not realize when the rules had exceptions, fail to see the times when the usual method won't cut through.

On August 26, 1346, a smaller contingent of English soldiers met a numerically superior French force outside of the Norman village of Crecy. The outnumbered English focused primarily on defense, using mainly dismounted spear- and longbowmen, whereas the French employed a highly mobile, offensive strategy utilizing mainly armored horsemen; the result was a decisive victory for the English, and a complete rout for the French. 10 years later, English and French forces would meet again at Poitiers; the French still had the numerical advantage, but their strategies were reversed: thistime, the French fielded dismount archers, while English charged in on horseback. The English still won, and even managed to capture the French king, forcing him to sign over a sizable chunk of his kingdom as ransom. What the French commanders failed to understand was that Cracy was a flat plain, where attacking the enemy was easy as looking around till you saw him, and then attacking in that direction, but where strong defensive positions where at a premium; thus the side that was best at defense would win. All thefrench saw was that the English won at Crecy, and so they tried to copy the english's winning tactics, without ever understanding why those tactics would be inappropriate to the hillier, more rugged territory around Poitiers.

Adhocracies seem specifically designed to avoid this problem at an institutional level.

The word is a portmanteau of the Latin ad hoc, meaning "for the purpose", and the suffix -cracy, from the ancient Greek kratein (κρατεῖν), meaning "to govern",

With characteristics such as those described in the Wikipedia article I linked, is it possible to institutionalize avoidance of institutional incompetence, such as what we described here?

Another lost purposes example: flow charts for programming.

When I was very young I was also very curious, but I sated my curiosity by telling myself that I would know it all when I was grown up. It wasn't my problem to be curious about these things, other people were handling them. Maybe Eliezer's classmates were thinking that this didn't really make sense, but they trusted the adults enough to do it anyways. Like how a lit lightbulb turns less mysterious and wonderful when you know that someone else already knows the scientific reason for it. All you need to do is follow the instructions and you'll get light.

"I wish I lived in an era where I could just tell my readers they have to thoroughly research something, without giving insult."

Is that not what this entire site is accomplishing?

This post actually kind of suggests that Eliezer is an Unfriendly AI.

That is, human beings do not have some specific ultimate goal that they are trying to achieve, and they do not try to make everything else fit in with that. This is why they do not destroy the world. Correspondingly, the fear that an AI would destroy the world is precisely the fear that it would have a goal like that.

So Eliezer's abhorrence to lost purposes makes him believe that an AI would be like him; just as he fanatically seeks his goals, thereby risking destroying the world, so he expects an AI to fanatically seek its goals and destroy the world.

I expect an AI to be more like normal humans than like Eliezer. It will not fanatically seek any goal, and it will not abhor lost purposes.

Many examples are given in the C2 Wiki article on Old Rules With Forgotten Reasons.

"More than anything, you must be thinking of carrying your movement through to cutting him. You must thoroughly research this." (Emphasis added.)

I wish I could upvote this post multiple times. Especially the "You must thoroughly research this" part, although also all the examples of institutions failing to accomplish much of anything, even when you might naively have really thought they would (e.g., individual health procedures work; how could large-scale institutions for doing those procedures not work?). I read this post back in 2008 and was like "oh, cool, it's a nice handle for a common little bug that happens to me and other people". I reread it just now, and was more like "Shit, good point, we actually do need to 'thoroughly research' how to avoid lost purposes, or how to carry our movements through to cutting the enemy. For lack of this, the EA and AI Safety movements have perhaps mostly wasted their last several years." Recommending the "thoroughly research" part to anyone who wants to carry on a complex task across a set of tens or hundreds (or thousands) of people without having all the value dissipate into noise.

Interesting. It seems your early Jewish teachers were firmly stuck in the mindset that was one of the things Christianity originally arose to combat. (Not that Christians should feel too haughty about that since the vast majority of Christian sects fell right back into the same trap early on. I find it amusing that "The Lord's Prayer" is repeated by-rote in most sects when in-context it was given as an example of how you shouldn't waste your time with by-rote prayers...)

This is as opposed to the Jewish community in and around Salt Lake City in the early 1900s that my grandfather always told stories about. (Although I must admit the possibility that the stories were modified to better carry whatever principle he was trying to teach at the expense of strict conformance to history.) To them the best prayers were actions. Prayers-of-words were useful only insofar as they led to choosing the best prayers-of-action. If your neighbour's barn burns down you don't stand around muttering prayers that they'll be able to recover from the disaster any longer than it takes you to figure out who can contribute what -- and then you go out, build them a new barn and stock it for them. Prayers-of-words-you-don't-understand are worse than useless. They are a distraction that leads you in entirely the wrong direction. Prayers-of-words-that-don't-cause-action are empty and useless -- intermediate steps detached from their terminal goals. But we lazy humans are creatures of ritual and habit and pattern-matching. If we experience often enough a chain of events that starts with words and ends with desirable outcomes we eventually become tempted to think that all we need to do is start the chain and not give any thought to the middle steps. If every day at 4PM we take our keys and get in the car and drive to the store and buy some chocolate we may well find ourselves wandering the aisles aimlessly despite having heard about the chocolate-destroying-earthquake prior to having left the house. Because we weren't actually thinking about it.

"Things we do without thinking -- There's the real danger."

"Likewise with people not automatically noticing [? · GW] when supposedly selfish people give altruistic arguments in favor of selfishness, or when supposedly altruistic people give selfish arguments in favor of altruism" - maybe it's because they want to encompass everyone? So the altruistic guy thinks "other altruistic guys like myself are not to be encourage to be altruistic but I need to appeal somehow to those who are currently selfish", and vice versa?

Dead link for "Why Most Published Research Findings Are False". Googling just the url parameters yields this.